Evaluation of Nine Consensus Indices in Delphi Foresight Research and Their Dependency on Delphi Survey Characteristics: A Simulation Study and Debate on Delphi Design and Interpretation
نویسندگان
چکیده
The extent of consensus (or the lack thereof) among experts in emerging fields of innovation can serve as antecedents of scientific, societal, investor and stakeholder synergy or conflict. Naturally, how we measure consensus is of great importance to science and technology strategic foresight. The Delphi methodology is a widely used anonymous survey technique to evaluate consensus among a panel of experts. Surprisingly, there is little guidance on how indices of consensus can be influenced by parameters of the Delphi survey itself. We simulated a classic three-round Delphi survey building on the concept of clustered consensus/dissensus. We evaluated three study characteristics that are pertinent for design of Delphi foresight research: (1) the number of survey questions, (2) the sample size, and (3) the extent to which experts conform to group opinion (the Group Conformity Index) in a Delphi study. Their impacts on the following nine Delphi consensus indices were then examined in 1000 simulations: Clustered Mode, Clustered Pairwise Agreement, Conger's Kappa, De Moivre index, Extremities Version of the Clustered Pairwise Agreement, Fleiss' Kappa, Mode, the Interquartile Range and Pairwise Agreement. The dependency of a consensus index on the Delphi survey characteristics was expressed from 0.000 (no dependency) to 1.000 (full dependency). The number of questions (range: 6 to 40) in a survey did not have a notable impact whereby the dependency values remained below 0.030. The variation in sample size (range: 6 to 50) displayed the top three impacts for the Interquartile Range, the Clustered Mode and the Mode (dependency = 0.396, 0.130, 0.116, respectively). The Group Conformity Index, a construct akin to measuring stubbornness/flexibility of experts' opinions, greatly impacted all nine Delphi consensus indices (dependency = 0.200 to 0.504), except the Extremity CPWA and the Interquartile Range that were impacted only beyond the first decimal point (dependency = 0.087 and 0.083, respectively). Scholars in technology design, foresight research and future(s) studies might consider these new findings in strategic planning of Delphi studies, for example, in rational choice of consensus indices and sample size, or accounting for confounding factors such as experts' variable degrees of conformity (stubbornness/flexibility) in modifying their opinions.
منابع مشابه
Methodological Principles and Applications of the Delphi Method: A Narrative Review
Background and Objectives: The appearance of complex issues with insufficient information has resulted in the consensus or unanimity spread. The consensus methods include the nominal group and the Delphi method.The Delphi method is a systematic process to predict and help making decision through survey rounds and information gathering and finally the group consensus. This method has been design...
متن کاملDesigning Brilliant Model for Human Resources Excellence in Industry with Fuzzy Delphi Technique
The aim of this research is to design a model for excellence in human resources emphasizing its results in organizational behavior. This research depending on the purpose of basic research is also based on the research project and the terms of data collection, research is a descriptive (non-experimental), exploratory and adaptive. The instrument used in this study is a questionnaire consisting ...
متن کاملThe Investigation of the Perspectives of Iranian EFL Domain Experts on Postmethod Pedagogy: A Delphi Technique
After the introduction of postmethod pedagogy by Kumaravadivelu with its three principles of particularity, possibility and practicality, a wave of attention was directed towards this so-called 'postmethod era' and its appropriacy and adequacy in satiating the demands of the language learners in this 'brand new world'. This situation has created a healthy debate among the Iranian EFL community ...
متن کاملAn Introduction to Policy Delphi; A tool for discovering the opposing views on health policy issues
Objective: In this review, we investigated various aspects of Policy Delphi technique to make decision-makers more aware of this pertinent method so that they can use it in their policy decisions in their organizations. Information sources and selected methods for study: This study was conducted using a review method and by searching the related literature in databases such as PubMed, Scopus a...
متن کاملExamine poverty alleviation policies in Iran based on overwhelming documents
The main purpose of the present study is to identify effective strategies to reduce poverty based on extracurricular documents with regional foresight approach. The method of this study is descriptive and qualitative in terms of research propositions. It has been done in two stages of library and Delphi method and with reference to 37 academics and executives in economics and management. In the...
متن کامل